1. Field of the Invention
The present invention relates generally to an electronic musical performance system that simplifies the playing of music, and more particularly, to methods and systems for using traditional music gestures to control the playing of music.
2. Description of the Prior Art
TRADITIONAL MUSICAL INSTRUMENTS
Musical instruments have traditionally been difficult to play. To play an instrument a student must simultaneously control pitch, timbre (sound quality), and rhythm. To play in an ensemble, the student must also keep in time with the other musicians. Some instruments, such as the violin, require a considerable investment of time to develop enough mechanical skill and technique to produce a single note of acceptable timbre. Typically a music student will start with simple, often uninspiring, music.
Once a musician becomes proficient at playing a sequence of notes in proper pitch, timbre, and rhythm, the musician can start to develop the skills of expression. Slight variations in the timing of notes, called rubato, and the large scale speeding and slowing called tempo are both temporal methods of bringing life to a musical score. Variations of volume and timbre also contribute to the expression of a musical piece. Musical expression distinguishes a technically accurate, yet dry, rendition of a piece of music from an exciting and moving interpretation. In both instances the correct sequence of notes as specified in a musical score are played, but in the latter, the musician, through manipulation of timing and timbre, has brought out the expressive meaning of the piece which is not fully defined in the score.
For those people who want to experience the pleasures of playing a musical instrument but do not have the necessary training, technique, and skills, they must postpone their enjoyment and endure arduous practice and music lessons. The same applies for those who want to play with others but are not proficient enough to play the correct note at the correct volume, time, and timbre, fast enough to keep up with the others. Many beginning music students abandon their study of music along the way when faced with the frustration and demands of learning to play a musical instrument.
ELECTRONIC MUSIC CONTROLLERS
The introduction of electronic music technology, however, has made a significant impact on students participation in music. A music synthesizer, such as the Proteus from E-mu Systems of Santa Cruz, Calif., allows a novice keyboard player to control a variety of instrument sounds, including flute, trumpet, violin, and saxophone. With the standardization of an electrical interface protocol, Musical Instrument Digital Interface (MIDI), it is now possible to connect a variety of controllers to a synthesizer.
A controller is a device that sends commands to a music synthesizer, instructing the synthesizer to generate sounds. A wide variety of commercially available controllers exist and can be categorized as traditional and alternative. Traditional controllers are typically musical instruments that have been instrumented to convert the pitch of the instrument into MIDI commands. Examples of traditional controllers include the violin, cello, and guitar controllers by Zeta Systems (Oakland, Calif.); Softwind's Synthaphone saxophone controller; the stringless fingerboard synthesizer controller, U.S. Pat. No. 5,140,887, dated Aug. 25, 1992, issued to Emmett Chapman; the digital high speed guitar synthesizer, U.S. Pat. No. 4,336,734, dated Jun. 29, 1982, issued to Robert D. Polson; and the electronic musical instrument with quantized resistance strings, U.S. Pat. No. 4,953,439, dated Sep. 4, 1990, issued to Harold R. Newell.
A technology which is an integral part of many traditional controllers is a pitch tracker, a device which extracts the fundamental pitch of a sound. IVL Technologies of Victoria, Canada manufactures a variety of pitch-to-MIDI interfaces, including The Pitchrider 4000 for wind and brass instruments; Pitchrider 7000 for guitars; and Steelrider, for steel guitars.
Some traditional controllers are fully electronic, do not produce any natural acoustic sound, and must be played with a music synthesizer. They typically are a collection of sensors in an assembly designed to look and play like the instrument they model. Commercial examples of the non-acoustic traditional controllers which emulate wind instruments include Casio's DH-100 Digital Saxophone controller, Yamaha's WXll and Windjamm'r wind instrument controller, and Akai's WE1000 wind controller. These controllers sense the closing of switches to determine the pitch intended by the player.
Alternative controllers are sensors in a system that typically control music in an unconventional way. One of the earliest, pre-MIDI, examples is the Theremin controller where a person controlled the pitch and amplitude of a tone by the proximity of their hands to two antenna. Some examples of alternative controllers include Thunder (trademark), a series of pressure pads controlled by touch, and Lightening (trademark), a system in which you wiggle an infrared light in front of sensors, both developed and Sold by Don Buchla and Associates (Berkeley, Calif.); Videoharp, a controller that optically tracks fingertips, by Dean Rubine and Paul McAvinney of Carnegie-Mellon University; Biomuse, a controller that senses and processes brain waves and muscle activity (electromyogram), by R. Benjamin Knapp of San Jose State University and Hugh S. Lusted of Stanford University; Radio Drum, a three dimensional baton and gesture sensor, U.S. Pat. No. 4,980,519, dated Dec. 25, 1990, issued to Max V. Mathews; and a music tone control apparatus which measures finger bending, U.S. Pat. No. 5,125,313, dated Jun. 30, 1992, issued to Teruo Hiyoshi, et al.
The traditional controllers enable a musician skilled on one instrument to play another. For example, a saxophonist using Softwind's Synthaphone saxophone controller can control a synthesizer set to play the timbre of a flute. Cross-playing becomes difficult when the playing technique of the controller does not convert well to the timbre to be played. For example a saxophonist trying to control a piano timbre will have difficulty playing a chord since a saxophone is inherently monophonic. A more subtle difference is a saxophonist trying to control a violin. How does the saxophonist convey different bowing techniques such as reversal of bow direction (detache and legato), the application of significant bow pressure before bow movement (martele, marcato, and staccato), and dropped, lifted or ricocheted strokes of the bow (pique, spiccato, jete and flying staccato). Conventional violin controllers do not make sufficient measurements of bow contact, pressure, and velocity to respond to these bowing techniques. To do so would encumber the playablity of the instrument or affect its ability to produce a good quality acoustic signal. However, these bow gestures have an important effect on the timbre of sound and are used to convey expression to music.
Tod Machover and his students at M.I.T. have been extending the playing technique of traditional musical instruments by applying sensors to acoustic instruments and connecting them to computers (Machover, T., "Hyperinstrument, A Progress Report 1997-1991", MIT Media Laboratory, January 1992). These extended instruments, called hyperinstruments, allow a trained musicians to experiment with new ways of manipulating synthesized sound. Once such instrument, the Hyperlead Guitars, the timbre of a sequence of notes played by a synthesizer is controlled by the position of the guitarist's hand on the fret board. In another implementation, the notes of guitar chords automatically selected from a score stored inside a computer, are assigned to the strings of a guitar. Picking a string triggers the note assigned to the string, with a timbre determined by fret position. Neither of these implementations allows traditional guitar playing technique where notes are triggered by either hand.
EASY-TO-PLAY MUSICAL ACOUSTIC INSTRUMENTS
Musical instruments have been developed that simplify the production of sound by limiting the pitches that can be produced. The autoharp is a harp with racks of dampers that selectively mute strings of un-desired pitch, typically those not belonging to a particular chord. A harmonica is a series of vibrating reeds of selected pitches. Toy xylophones and piano exists that only have the pitches of a major scale.
VOICE CONTROLLED SYNTHESIZER
Marcian Hoff in U.S. Pat. NO. 4,771,671, dated Sep. 20, 1988, discloses an electronic music instrument that controls the pitch of a music synthesizer with the pitch of a human voice, later manufactured as the Vocalizer by Breakaway Systems (San Mateo, Calif.). The Vocalizer limits pitches to selected ones, similar to an autoharp. The Vocalizer includes a musical accompaniment which dynamically determines which pitches are allowed. If the singer produces a pitch that is not allowed, the device selects and plays the closest allowable pitch.
The difficulty in adopting Hoff's method to play a musical melody is that a vocalized pitch must be produced for each note played. Fast passages of music would require considerable skill of the singer to produce distinct and recognizable pitches. Such passages would also make great demands of the system to distinguish the beginning and ending of note utterances. The system has the same control problems as a saxophone controller mentioned above: singing technique does not convert well to controlling other instruments. For example, how does one strum a guitar or distinguish between bowing and plucking a violin with a voice controller?
ACCOMPANIMENT SYSTEMS
Accompaniment systems exist that allow a musician to sing or play along with a pre-recorded accompaniment. For the vocalist, karaoke is the use of a predefined, usually prerecorded, musical background to supply contextual music around which a person sings a lead part. Karaoke provides an enjoyable way to learn singing technique and is a form of entertainment. For the instrumentalist, a similar concept of "music-minus-one" exists, where, typically, the lead part of a musical orchestration is absent. Hundreds of classical and popular music titles exist for both karaoke and music-minus-one. Both concepts require the user to produce the correct sequence of notes, with either their voice or their instrument, to play the melody.
Musical accompaniment also exists on electronic keyboards and organs, from manufacturers such as Wurlitzer, Baldwin, Casio, and Yamaha, which allow a beginner to play a simple melody with an automatic accompaniment, complete with bass, drums, and chord changes.
A more sophisticated accompaniment method has been designed independently by Barry Vercoe (Vercoe, B., Puckette, M., "Synthetic Rehearsal: Training the Synthetic Performer", ICMC 1985 Proceedings, pages 275-278; Boulanger, R., "Conducting the MIDI Orchestra, Part 1", Computer Music Journal, Vol. 14, No. 2, Summer 1990, pages 39-42) and Roger Dannenberg (ibid., pages 42-46). Unlike previous accompaniment schemes where the musician follows the tempo of the accompaniment, they use the computer accompaniment to follow the tempo of the live musician by monitoring the notes played by the musician and comparing it to a score stored in memory. In Vercoe's system a flute and a violin were used as the melody instruments. In Dannenberg's system a trumpet was used.
In all of the cases of accompaniment mentioned, the person who plays the melody must still be a musician, having enough skill and technique to produce the proper sequence of pitches at the correct times and, where the instrument allows, with acceptable timbre, volume, and other expressive qualities.
SYSTEMS WITH STORED MELODY
In order to reduce the simultaneous tasks a person playing music must perform, a music re-performance system can store a sequence of pitches, and through the action of the player, output these pitches. A toy musical instrument is described in U.S. Pat. No. 4,981,457, by Taichi Iimura et al, where the closing of a switch by a moveable part of the toy musical instrument is used to play the next note of a song stored in memory. Shaped like a violin or a slide trombone, the musical toy is an attempt to give the feeling of playing the instrument the toy imitates. The switch is closed by moving a bow across the bridge, for the violin, or sliding a slide tube, for the trombone. The length of each note is determined by the length of time the switch is closed, and the interval between notes is determined by the interval between switch closing. No other information is communicated from the controller to the music synthesizer.
The toy's limited controller sensor, a single switch makes playing fast notes difficult, limiting expression to note timing, and does not accommodate any violin playing technique that depends on bow placement, pressure, or velocity, and finger placement and pressure. Similarly the toy does not accommodate any trombone playing techniques that depends on slide placement, lip tension, or air pressure. The limited capability of the toy presents a fixed level of complexity to the player which, once surpassed, renders the toy boring.
The melody for a song stored in the toy's memory has no timing information, making it impossible for the toy to play the song itself, to provide guidance for the student, and does not contain any means to provide any synchronized accompaniment. The toy plays monophonic music while a violin, having four strings, polyphonic. The toy has no way to deal with a melody that starts a note before finishing the last, or ornamentations a player might add to a re-performance, such as playing a single long note as a series of shorter notes.
Another system that simplifies the tasks of the person playing music is presented by Max Mathews in his Conductor Program (Mathews, M. and Pierce, J., editors, "The Conductor Program and Mechanical Baton", Current Directions in Computer Music Research, The MIT Press, 1989, Chapter 19; Boulanger, R., "Conducting the MIDI Orchestra, Part 1", Computer Music Journal, Vol. 14, No. 2, Summer 1990, page 34-39). In Mathews' system a person conducts a score, which is stored in computer memory, using special batons, referred to earlier as the alternative controller Radio Drum.
Mathews' system is basically a musical sequencer with synchronization markers distributed through the score. The sequencer plays the notes of the score at the times specified, while monitoring the actions of the batons. If the sequencer reaches a synchronization marker before a baton gesture, the sequencer stops the music and waits for a gesture. If the baton gesture comes in advance of the marker, the sequencer jumps ahead to the next synchronization marker, dropping the notes in between. The system does not tolerate any lapses of attention by the performer. An extra beat can eliminate a multitude of notes. A missed beat will stop the re-performance.
Expressive controls of timbre, volume, pitch bend are controlled by a combination of spatial positions of the batons, joystick and knobs. Designed primarily as a control device for the tempo and synchronization of an accompaniment score, there are no provisions for controlling the relative timing of musical voices in the score. The controller is a cross between a conductor's baton and a drum mallet and does not use the gestures and playing techniques of the instruments being played. There is no way for several people to take part in the re-performance of music. Mathews' conductor system is a solo effort with no means to include any other players.
None of the systems and techniques presented that are accessible to non-musicians provides an adequate visceral and expressive playing experience of the instrument sounds they control. The natural gestural language people learn and expect from watching instruments being played are not sufficiently utilized, accommodated, or exploited in any of these system.
MIDI SEQUENCERS
With the advent of standardization of the electronic music interface, MIDI, many software application programs called sequencers became available to record, store, manipulate, and playback music. Commercial examples include Cakewalk by Twelve Tone Systems and Vision by Opcode Systems. One manipulation technique common to most sequencers is the ability to change the time and duration of notes. One such method is described in U.S. Pat. No. 4,969,384, by Shingo Kawasaki, et al., where the duration of individual sections of music can be shortened or lengthened.
Music can be input into sequencers by typing in notes and durations, drawing them in using a mouse pointing device, or more commonly, using the sequencer as a tape recorder and "playing live". For those not proficient at playing keyboard it is often difficult to play the correct sequence of notes at the correct time, with the correct volume. It is possible to "play in" the correct notes without regard for time and edit the time information later. This can be quite tedious as note timing is edited "off line", that is non-real time, yet music is only perceived while it is being played. Typically this involves repeatedly playing and editing the music in small sections, making adjustments to the location and duration of notes. Usually the end result is stilted for it is difficult to "edit-in" the feel of a piece of music.
It is therefore desirable to have a music editing system where selected music parameters (e.g. volume, note timing, timbre) can be altered by a musician re-playing the piece. Such a system, called a music re-performance system, would allow a musician to focus on the selected parameters being edited.